• 专利附录
  • 专利说明
  • 权利要求
  • 类似技术
  • 同族专利
  • 引用文献
  • 法律状态
  • 优先权
    • 专利摘要:
    A learning method for acquiring at least one personalized reward function, used for performing a Reinforcement Learning(RL) algorithm, corresponding to a personalized optimal policy for a subject driver is provided. And the method includes steps of: (a) a learning device performing a process of instructing an adjustment reward network to generate first adjustment rewards, by referring to the information on actual actions and actual circumstance vectors in driving trajectories, a process of instructing a common reward module to generate first common rewards by referring to the actual actions and the actual circumstance vectors, and a process of instructing an estimation network to generate actual prospective values by referring to the actual circumstance vectors; and (b) the learning device instructing a first loss layer to generate an adjustment reward and to perform backpropagation to learn parameters of the adjustment reward network.
    公开号:EP3690769A1
    申请号:EP20152797.5
    申请日:2020-01-21
    公开日:2020-08-05
    发明作者:Kye-hyeon KIM；Yongjoong Kim；Hak-Kyoung Kim；Woonhyun Nam；Sukhoon Boo；Myungchul Sung；Dongsoo Shin；Donghun Yeo；Wooju RYU；Myeong-Chun Lee；Hyungsoo Lee；Taewoong Jang；Kyungjoong Jeong；Hongmo Je；Hojin Cho
    申请人:Stradvision Inc；
    IPC主号:G06Q10-00
    专利说明:
    [0001] The present disclosure relates to a learning method and a learning device for use with an autonomous vehicle; and more particularly, to the learning method and the learning device for supporting a Reinforcement Learning(RL) by using human driving data as training data, to thereby perform a personalized path planning, and a testing method and a testing device using the same.
    [0002] An autonomous driving is performed to transport its passengers safely and quickly. However, sometimes, when a path planning is performed, what the autonomous driving endeavors to achieve and what the passengers want may be different.
    [0003] For example, some of the passengers may want a comfortable driving experience without a sudden stop or a sudden acceleration, rather than a quick but unstable driving experience. In this case, if the path planning performed by an autonomous vehicle carrying said some of the passengers allows the autonomous vehicle to drive quickly but not stably, said some of the passengers may be dissatisfied with an autonomous driving performed by the autonomous vehicle.
    [0004] Thus, it is important to personalize the path planning passenger by passenger, but it has not been studied much so far.
    [0005] It is an object of the present disclosure to solve all the aforementioned problems.
    [0006] It is another object of the present disclosure to provide a learning method for supporting a Reinforcement Learning(RL) algorithm by using human driving data as training data, to thereby provide a personalized path planning, and then to thereby provide a satisfied driving experience to passengers of an autonomous vehicle.
    [0007] It is still another object of the present disclosure to provide a personalized reward function to be used for supporting the RL algorithm, by using the human driving data as the training data, to thereby provide the personalized path planning.
    [0008] It is still yet another object of the present disclosure to provide a method for acquiring the personalized reward function by adjusting a common reward function to reduce usage of computing resources.
    [0009] In accordance with one aspect of the present disclosure, there is provided a learning method for acquiring at least one personalized reward function, used for performing a Reinforcement Learning(RL) algorithm, corresponding to a personalized optimal policy for a subject driver which is acquired by adjusting a common optimal policy established according to a common criterion for an autonomous driving, to thereby support the autonomous driving of a subject vehicle, including steps of: (a) a learning device, if one or more actual circumstance vectors and information on one or more actual actions performed at timings corresponding to the actual circumstance vectors by referring to one or more actual circumstances, corresponding thereto, included in each of one or more driving trajectories of the subject driver are acquired, performing (i) a process of instructing an adjustment reward network, which is built to operate as an adjustment reward function to be used for establishing the personalized reward function from a common reward function corresponding to the common optimal policy, to generate each of one or more first adjustment rewards corresponding to each of the actual actions performed at each of the timings, by referring to the information on the actual actions and the actual circumstance vectors, (ii) a process of instructing a common reward module corresponding to the common reward function to generate each of one or more first common rewards corresponding to each of the actual actions performed at each of the timings, by referring to the information on the actual actions and the actual circumstance vectors, and (iii) a process of instructing an estimation network, which is capable of estimating sums of personalized rewards generated while common optimal actions according to the common optimal policy are performed based on its corresponding actual circumstances, to generate each of one or more actual prospective values corresponding to each of the actual circumstances at each of the timings in the driving trajectories, by referring to the actual circumstance vectors; and (b) the learning device instructing a first loss layer to generate at least one adjustment reward loss by referring to (i) each of first personalized rewards corresponding to each of the first adjustment rewards and each of the first common rewards and (ii) the actual prospective values, and to perform backpropagation by referring to the adjustment reward loss, to thereby learn at least part of parameters of the adjustment reward network.
    [0010] As one example, at the step of (b), the learning device instructs the first loss layer to generate the adjustment reward loss by referring to a following formula:
    [0011] As one example, the method further includes steps of: (c) the learning device performing (i) a process of instructing the adjustment reward network to generate one or more second adjustment rewards corresponding to each of the common optimal actions to be performed at each of the timings of the driving trajectories by referring to the actual circumstance vectors, (ii) a process of instructing the common reward module to generate one or more second common rewards corresponding to each of the common optimal actions to be performed at each of the timings of the driving trajectories by referring to the actual circumstance vectors, and (iii) a process of instructing the estimation network, by referring to each of one or more virtual circumstance vectors corresponding to each of virtual circumstances caused by performing the common optimal actions at each of the timings of the driving trajectories, to generate one or more virtual prospective values corresponding to the virtual circumstances; and (d) the learning device instructing a second loss layer to generate at least one estimation loss by referring to (i) each of second personalized rewards corresponding to each of the second adjustment rewards and each of the second common rewards, (ii) the virtual prospective values and (iii) the actual prospective values, and to perform backpropagation by referring to the estimation loss, to thereby learn at least part of parameters of the estimation network.
    [0012] As one example, at the step of (d), the learning device instructs the second loss layer to generate the estimation loss by referring to a following formula:
    [0013] As one example, the virtual circumstance vectors are acquired by applying a circumstance estimation operation to each of at least part of the actual circumstance vectors and its corresponding common optimal action corresponding to the common optimal policy, wherein the circumstance estimation operation is performed by a pre-trained circumstance expectation network or performed by (i) instructing a virtual world simulator to simulate a specific actual circumstance corresponding to a specific actual circumstance vector in a virtual world, (ii) instructing a virtual vehicle in the specific actual circumstance to perform one of the common optimal actions according to the common optimal policy, and (iii) detecting changes of the virtual world caused by said one of the common optimal actions.
    [0014] As one example, the learning device repeats a process of training the adjustment reward network corresponding to the step of (a) and the step of (b) and a process of training the estimation network corresponding to the step of (c) and the step of (d), to thereby fully train the adjustment reward network and the estimation network.
    [0015] As one example, the driving trajectories are provided to the learning device as a mini batch generated by randomly sampling the driving trajectories from a driving trajectory group corresponding to the subject driver.
    [0016] As one example, the common optimal actions according to the common optimal policy are determined by a general RL agent optimized by performing the RL algorithm using the common reward module corresponding to the common optimal policy.
    [0017] In accordance with another aspect of the present disclosure, there is provided a testing method for using at least one personalized reward function to train a personalized Reinforcement Learning(RL) agent, corresponding to a personalized optimal policy for a subject driver which is acquired by adjusting a common optimal policy established according to a common criterion for an autonomous driving, to thereby support the autonomous driving of a subject vehicle, including steps of: (a) on condition that (1) a learning device, if one or more actual circumstance vectors for training and information on one or more actual actions for training performed at timings for training corresponding to the actual circumstance vectors for training by referring to one or more actual circumstances for training, corresponding thereto, included in each of one or more driving trajectories for training of the subject driver are acquired, has performed (i) a process of instructing an adjustment reward network, which is built to operate as an adjustment reward function to be used for establishing the personalized reward function from a common reward function corresponding to the common optimal policy, to generate each of one or more first adjustment rewards for training corresponding to each of the actual actions for training performed at each of the timings for training, by referring to the information on the actual actions for training and the actual circumstance vectors for training, (ii) a process of instructing a common reward module corresponding to the common reward function to generate each of one or more first common rewards for training corresponding to each of the actual actions for training performed at each of the timings for training, by referring to the information on the actual actions for training and the actual circumstance vectors for training, and (iii) a process of instructing an estimation network, which is capable of estimating sums of personalized rewards for training generated while common optimal actions for training according to the common optimal policy for training are performed based on its corresponding actual circumstances for training, to generate each of one or more actual prospective values for training corresponding to each of the actual circumstances for training at each of the timings for training in the driving trajectories for training, by referring to the actual circumstance vectors for training; and (2) the learning device has instructed a first loss layer to generate at least one adjustment reward loss by referring to (i) each of first personalized rewards for training corresponding to each of the first adjustment rewards for training and each of the first common rewards for training and (ii) the actual prospective values for training, and to perform backpropagation by referring to the adjustment reward loss, to thereby learn at least part of parameters of the adjustment reward network, a testing device instructing the adjustment reward network and the common reward module to generate a personalized reward for testing including an adjustment reward for testing and a common reward for testing by referring to (i) an actual circumstance vector for testing corresponding to a t-th timing and (ii) an actual action for testing generated by the personalized RL agent; (b) the testing device instructing the personalized RL agent to train its own parameters by referring to the personalized reward for testing.
    [0018] As one example, at the step of (b), the personalized RL agent trains said its own parameters by referring to the personalized reward for testing, to thereby support the subject vehicle to drive similarly to the actual actions for training.
    [0019] In accordance with still another aspect of the present disclosure, there is provided a learning device for acquiring at least one personalized reward function, used for performing a Reinforcement Learning(RL) algorithm, corresponding to a personalized optimal policy for a subject driver which is acquired by adjusting a common optimal policy established according to a common criterion for an autonomous driving, to thereby support the autonomous driving of a subject vehicle, including: at least one memory that stores instructions; and at least one processor configured to execute the instructions to perform processes of: (I) if one or more actual circumstance vectors and information on one or more actual actions performed at timings corresponding to the actual circumstance vectors by referring to one or more actual circumstances, corresponding thereto, included in each of one or more driving trajectories of the subject driver are acquired, performing (i) a process of instructing an adjustment reward network, which is built to operate as an adjustment reward function to be used for establishing the personalized reward function from a common reward function corresponding to the common optimal policy, to generate each of one or more first adjustment rewards corresponding to each of the actual actions performed at each of the timings, by referring to the information on the actual actions and the actual circumstance vectors, (ii) a process of instructing a common reward module corresponding to the common reward function to generate each of one or more first common rewards corresponding to each of the actual actions performed at each of the timings, by referring to the information on the actual actions and the actual circumstance vectors, and (iii) a process of instructing an estimation network, which is capable of estimating sums of personalized rewards generated while common optimal actions according to the common optimal policy are performed based on its corresponding actual circumstances, to generate each of one or more actual prospective values corresponding to each of the actual circumstances at each of the timings in the driving trajectories, by referring to the actual circumstance vectors; and (II) instructing a first loss layer to generate at least one adjustment reward loss by referring to (i) each of first personalized rewards corresponding to each of the first adjustment rewards and each of the first common rewards and (ii) the actual prospective values, and to perform backpropagation by referring to the adjustment reward loss, to thereby learn at least part of parameters of the adjustment reward network.
    [0020] As one example, at the process of (II), the processor instructs the first loss layer to generate the adjustment reward loss by referring to a following formula:
    [0021] As one example, wherein the processor further performs a process of: (III) performing (i) a process of instructing the adjustment reward network to generate one or more second adjustment rewards corresponding to each of the common optimal actions to be performed at each of the timings of the driving trajectories by referring to the actual circumstance vectors, (ii) a process of instructing the common reward module to generate one or more second common rewards corresponding to each of the common optimal actions to be performed at each of the timings of the driving trajectories by referring to the actual circumstance vectors, and (iii) a process of instructing the estimation network, by referring to each of one or more virtual circumstance vectors corresponding to each of virtual circumstances caused by performing the common optimal actions at each of the timings of the driving trajectories, to generate one or more virtual prospective values corresponding to the virtual circumstances; and (IV) instructing a second loss layer to generate at least one estimation loss by referring to (i) each of second personalized rewards corresponding to each of the second adjustment rewards and each of the second common rewards, (ii) the virtual prospective values and (iii) the actual prospective values, and to perform backpropagation by referring to the estimation loss, to thereby learn at least part of parameters of the estimation network.
    [0022] As one example, at the process of (IV), the processor instructs the second loss layer to generate the estimation loss by referring to a following formula:
    [0023] As one example, the virtual circumstance vectors are acquired by applying a circumstance estimation operation to each of at least part of the actual circumstance vectors and its corresponding common optimal action corresponding to the common optimal policy, wherein the circumstance estimation operation is performed by a pre-trained circumstance expectation network or performed by (i) instructing a virtual world simulator to simulate a specific actual circumstance corresponding to a specific actual circumstance vector in a virtual world, (ii) instructing a virtual vehicle in the specific actual circumstance to perform one of the common optimal actions according to the common optimal policy, and (iii) detecting changes of the virtual world caused by said one of the common optimal actions.
    [0024] As one example, the processor repeats a process of training the adjustment reward network corresponding to the process of (I) and the process of (II) and a process of training the estimation network corresponding to the process of (III) and the process of (IV), to thereby fully train the adjustment reward network and the estimation network.
    [0025] As one example, the driving trajectories are provided to the learning device as a mini batch generated by randomly sampling the driving trajectories from a driving trajectory group corresponding to the subject driver.
    [0026] As one example, the common optimal actions according to the common optimal policy are determined by a general RL agent optimized by performing the RL algorithm using the common reward module corresponding to the common optimal policy.
    [0027] In accordance with still yet another aspect of the present disclosure, there is provided a testing device for using at least one personalized reward function to train a personalized Reinforcement Learning(RL) agent, corresponding to a personalized optimal policy for a subject driver which is acquired by adjusting a common optimal policy established according to a common criterion for an autonomous driving, to thereby support the autonomous driving of a subject vehicle, including: at least one memory that stores instructions; and at least one processor configured to execute the instructions to perform processes of: (I) on condition that (1) a learning device, if one or more actual circumstance vectors for training and information on one or more actual actions for training performed at timings for training corresponding to the actual circumstance vectors for training by referring to one or more actual circumstances for training, corresponding thereto, included in each of one or more driving trajectories for training of the subject driver are acquired, has performed (i) a process of instructing an adjustment reward network, which is built to operate as an adjustment reward function to be used for establishing the personalized reward function from a common reward function corresponding to the common optimal policy, to generate each of one or more first adjustment rewards for training corresponding to each of the actual actions for training performed at each of the timings for training, by referring to the information on the actual actions for training and the actual circumstance vectors for training, (ii) a process of instructing a common reward module corresponding to the common reward function to generate each of one or more first common rewards for training corresponding to each of the actual actions for training performed at each of the timings for training, by referring to the information on the actual actions for training and the actual circumstance vectors for training, and (iii) a process of instructing an estimation network, which is capable of estimating sums of personalized rewards for training generated while common optimal actions for training according to the common optimal policy for training are performed based on its corresponding actual circumstances for training, to generate each of one or more actual prospective values for training corresponding to each of the actual circumstances for training at each of the timings for training in the driving trajectories for training, by referring to the actual circumstance vectors for training; and (2) the learning device has instructed a first loss layer to generate at least one adjustment reward loss by referring to (i) each of first personalized rewards for training corresponding to each of the first adjustment rewards for training and each of the first common rewards for training and (ii) the actual prospective values for training, and to perform backpropagation by referring to the adjustment reward loss, to thereby learn at least part of parameters of the adjustment reward network, instructing the adjustment reward network and the common reward module to generate a personalized reward for testing including an adjustment reward for testing and a common reward for testing by referring to (i) an actual circumstance vector for testing corresponding to a t-th timing and (ii) an actual action for testing generated by the personalized RL agent; (II) instructing the personalized RL agent to train its own parameters by referring to the personalized reward for testing.
    [0028] As one example, at the process of (II), the personalized RL agent trains said its own parameters by referring to the personalized reward for testing, to thereby support the subject vehicle to drive similarly to the actual actions for training.
    [0029] In addition, recordable media that are readable by a computer for storing a computer program to execute the method of the present disclosure is further provided.
    [0030] The above and other objects and features of the present disclosure will become apparent from the following description of preferred embodiments given in conjunction with the accompanying drawings.
    [0031] The following drawings to be used to explain example embodiments of the present disclosure are only part of example embodiments of the present disclosure and other drawings can be obtained based on the drawings by those skilled in the art of the present disclosure without inventive work.
    [0032] Fig. 1 is a drawing schematically illustrating a configuration of a learning device performing a learning method for supporting a Reinforcement Learning(RL) by using human driving data as training data, to thereby perform a personalized path planning in accordance with one example embodiment of the present disclosure.
    [0033] Fig. 2 is a drawing schematically illustrating an example of driving trajectories used for performing the learning method for supporting the RL by using the human driving data as the training data, to thereby perform the personalized path planning in accordance with one example embodiment of the present disclosure.
    [0034] Fig. 3 is a drawing schematically illustrating a flow of the learning method for supporting the RL by using the human driving data as the training data, to thereby perform the personalized path planning in accordance with one example embodiment of the present disclosure.
    [0035] Detailed explanation on the present disclosure to be made below refer to attached drawings and diagrams illustrated as specific embodiment examples under which the present disclosure may be implemented to make clear of purposes, technical solutions, and advantages of the present disclosure. These embodiments are described in sufficient detail to enable those skilled in the art to practice the disclosure.
    [0036] Besides, in the detailed description and claims of the present disclosure, a term "include" and its variations are not intended to exclude other technical features, additions, components or steps. Other objects, benefits, and features of the present disclosure will be revealed to one skilled in the art, partially from the specification and partially from the implementation of the present disclosure. The following examples and drawings will be provided as examples but they are not intended to limit the present disclosure.
    [0037] Moreover, the present disclosure covers all possible combinations of example embodiments indicated in this specification. It is to be understood that the various embodiments of the present disclosure, although different, are not necessarily mutually exclusive. For example, a particular feature, structure, or characteristic described herein in connection with one embodiment may be implemented within other embodiments without departing from the spirit and scope of the present disclosure. In addition, it is to be understood that the position or arrangement of individual elements within each disclosed embodiment may be modified without departing from the spirit and scope of the present disclosure. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present disclosure is defined only by the appended claims, appropriately interpreted, along with the full range of equivalents to which the claims are entitled. In the drawings, like numerals refer to the same or similar functionality throughout the several views.
    [0038] Any images referred to in the present disclosure may include images related to any roads paved or unpaved, in which case the objects on the roads or near the roads may include vehicles, persons, animals, plants, buildings, flying objects like planes or drones, or any other obstacles which may appear in a road-related scene, but the scope of the present disclosure is not limited thereto. As another example, said any images referred to in the present disclosure may include images not related to any roads, such as images related to alleyway, land lots, sea, lakes, rivers, mountains, forests, deserts, sky, or any indoor space, in which case the objects in said any images may include vehicles, persons, animals, plants, buildings, flying objects like planes or drones, ships, amphibious planes or ships, or any other obstacles which may appear in a scene related to alleyway, land lots, sea, lakes, rivers, mountains, forests, deserts, sky, or any indoor space, but the scope of the present disclosure is not limited thereto.
    [0039] To allow those skilled in the art to the present disclosure to be carried out easily, the example embodiments of the present disclosure by referring to attached drawings will be explained in detail as shown below.
    [0040] For reference, in the description below, the phrase "for training" or "training" is added for terms related to the learning process, and the phrase "for testing" or "testing" is added for terms related to the testing process, to avoid possible confusion.
    [0041] Fig. 1 is a drawing schematically illustrating a configuration of a learning device performing a learning method for a reinforcement learning by using human driving data as training data, to thereby perform a personalized path planning in accordance with one example embodiment of the present disclosure.
    [0042] By referring to Fig. 1, the learning device may include an adjustment reward network 130, an estimation network 140, a first loss layer 150, a second loss layer 160 and a common reward module 170 to be described later. Processes of input/output and computations of the adjustment reward network 130, the estimation network 140, the first loss layer 150, the second loss layer 160 and the common reward module 170 may be respectively performed by at least one communication part 110 and at least one processor 120. However, detailed communication schematics between the communication part 110 and the processor 120 are omitted in Fig. 1. Herein, a memory 115 may have stored various instructions to be described later, and the processor 120 may execute the instructions stored in the memory 115 and may perform processes of the present disclosure by executing the instructions to be disclosed later. Such description of the learning device 100 does not exclude an integrated device including any combination of a processor, a memory, a medium, or any other computing components.
    [0043] The adjustment reward network 130 and the estimation network 140 may include each of multiple layers including virtual neurons which acquire their inputs from their previous virtual neurons, process the inputs, and deliver their outputs to their next virtual neurons. To be simple, the adjustment reward network 130 and the estimation network 140 may have structures similar to those of well-known Feed-Forward networks.
    [0044] So far the configuration of the learning device 100 performing the learning method of the present disclosure has been explained. Below, the learning method itself will be explained, but before that, general explanation on backgrounds thereof will be provided.
    [0045] First, as generally known, the RL algorithm is a scheme that an RL agent is trained by (i) selecting a specific action based on a specific circumstance, (ii) acquiring a specific reward for the specific action by using a reward function assigned to the RL agent, and (iii) performing backpropagation or other training schemes by using the specific reward. In order to acquire a proper output, a programmer should establish the reward function properly, since the reward function is a key for training the RL agent.
    [0046] On this basis, a common optimal policy may be an autonomous driving scheme established according to a common criterion. And, a common reward function may be a reward function for training the RL agent to perform the autonomous driving according to the common optimal policy.
    [0047] On the other hand, a personalized optimal policy may be an autonomous driving scheme established for a subject driver. And, a personalized reward function may be reward function for training the RL agent to perform the autonomous driving according to the personalized optimal policy.
    [0048] In this regard, the present disclosure is directed to a goal of providing the personalized reward function corresponding to the personalized optimal policy for the autonomous driving, by slightly adjusting the common reward function, using an adjustment reward function. Such relationship may be as shown below: R p = R common + R driver
    [0049] In the formula, Rcommon may denote an output of the common reward module 170 performing the common reward function corresponding to the common optimal policy for the autonomous driving, Rdriver may denote an output of the adjustment reward network 130 performing the adjustment reward function, Rp may denote an output of the personalized reward function, acquired by using the common reward module 170 and the adjustment reward network 130 together.
    [0050] Herein, the common reward function 170 may be given as a rule-set from the beginning, and the adjustment reward network 130 may be given as untrained at first, and may be trained by performing the learning method of the present disclosure. Someone may think that it may be possible to build a singular neural network performing the personalized reward function for the personalized optimal policy, not like the present disclosure using two components, i.e., the adjustment reward network 130 and the common reward module 170. However, the personalized reward function cannot be performed by using the singular neural network, because a solution of the personalized reward function is not unique, and too much training data, i.e., driving trajectories to be explained later, is required to train the singular neural network. Thus, the personalized reward function may be performed by using both the common reward module 170 and the adjustment reward network 130.
    [0051] Additional information on the common reward module 170 and the common optimal policy will be presented below. That is, the common reward module 170 may have been acquired by analyzing each of the driving trajectories of drivers including information on each of circumstances and information on actions of each of the drivers in each of the circumstances. For example, an annotator may determine whether each of the actions in the each of the driving trajectories caused accidents or not, may set each of rewards for each of the actions, may set a rule-set derived from relationships between said rewards and said actions as the common reward function, and may build a module including the rule-set as the common reward module 170. Such common reward module 170 may be used for supporting a training process of the RL agent to which the driving trajectories are inputted as the training data. As a result, the RL agent may become capable of performing the autonomous driving by referring to the common optimal policy.
    [0052] Herein, the common reward function performed by the common reward module 170 may be expressed as a following formula: R common S A S next
    [0053] Herein, S may denote one of the circumstances at each of timings in one of the driving trajectories, A may denote an action performed at its corresponding timing, and Snext may denote its next one of the circumstances caused by the action. The adjustment reward function may be noted as similarly to the common reward function.
    [0054] The driving trajectories mentioned so far may correspond to a plurality of people, thus may have been used for acquiring the common optimal policy. In the present disclosure, some of the driving trajectories, which correspond to a single person, i.e., the subject driver, will be used, because the present disclosure is directed to a method for acquiring the "personalized" optimal policy, not the "common" optimal policy. Accordingly, "driving trajectories" to be mentioned below all correspond to the single person, i.e., the subject driver.
    [0055] Herein, the driving trajectories for the subject driver may include information on one or more actual circumstance vectors and one or more actual actions performed at timings corresponding to the actual circumstance vectors by referring to the one or more actual circumstances corresponding thereto. To be explained later, additional information, i.e., virtual circumstance vectors which include information on each of virtual circumstances caused by virtually performing common optimal actions, not the actual actions, can be used along with the driving trajectories, at each of the timings of the driving trajectories. Herein, the actual circumstance vectors may include information on surroundings of the subject vehicle at their corresponding timings, such as information on locations of surrounding objects and their classes, or information on segmentation images. In order to take a look at such driving trajectories and the additional information, Fig. 2 will be referred to.
    [0056] Fig. 2 is a drawing schematically illustrating an example of driving trajectories used for performing the learning method for supporting the RL by using the human driving data as the training data, to thereby perform the personalized path planning in accordance with one example embodiment of the present disclosure.
    [0057] By referring to Fig. 2, notations s, a, s' and a' for circles and arrows can be seen. Herein, each of s and a may denote each of the actual circumstance vectors and each of the actual actions, and each of s' and a' may denote each of the virtual circumstance vectors and their corresponding common optimal actions. More specifically, s' may denote each of the virtual circumstance vectors caused by performing each of the common optimal actions a', not the actual actions a, at each of states corresponding to each of the actual circumstance vectors s.
    [0058] Herein, how the common optimal actions and the virtual circumstance vectors can be acquired will be explained. First, the common optimal actions can be acquired from the RL agent including the common optimal policy, by inputting the actual circumstance vectors in the driving trajectories to the RL agent. The virtual circumstance vectors may be acquired by using an additional operation, i.e., a circumstance expectation operation. And the circumstance expectation operation may be performed in two ways.
    [0059] First, the virtual circumstance vectors may be acquired by using a pre-trained circumstance expectation network. The circumstance expectation network may include each of multiple layers including multiple neurons. Such circumstance expectation network may acquire circumstance vectors for training and their corresponding actions for training as its training data, may output estimated next circumstance vectors for training, and may generate a loss by using the estimated next circumstance vectors for training and their corresponding Ground-Truth(GT) next circumstance vectors including information on circumstances caused by the actions for training in circumstances corresponding to the circumstance vectors for training. Then, the circumstance expectation network may perform backpropagation by using the loss, to thereby learn parameters thereof. Such training processes may be similar to those of a general Feed-Forward Network, thus a person in the art may be able to understand above explanations easily.
    [0060] Otherwise, the virtual circumstance vectors can be acquired by using a virtual world simulator. That is, the circumstance estimation operation may be performed by instructing the virtual world simulator to simulate a specific actual circumstance corresponding to a specific actual circumstance vector in a virtual world, instructing a virtual vehicle in the specific actual circumstance to perform one of the common optimal actions according to the common optimal policy, and detecting changes of the virtual world caused by said one of the common optimal actions, to thereby acquire the virtual circumstance vectors.
    [0061] The estimation network 140 will be explained briefly below. The estimation network 140 may acquire a specific circumstance vector corresponding to a specific timing as its input, and may output an estimated sum of personalized rewards, if the common optimal actions are continuously performed, to be generated from the specific timing to a final timing of its corresponding driving trajectory.
    [0062] By referring to general and brief explanations provided above, the learning method of the present disclosure will be explained by referring to Fig. 3.
    [0063] Fig. 3 is a drawing schematically illustrating a flow of the learning method for supporting the RL by using the human driving data as the training data, to thereby perform the personalized path planning in accordance with one example embodiment of the present disclosure.
    [0064] By referring to Fig. 3, at a step of S00, the learning device 100 may acquire the actual circumstance vectors and information on the actual actions performed at the timings corresponding to the actual circumstance vectors by referring to the actual circumstances, included in each of the driving trajectories of the subject driver. And, at a step of S01-1, the learning device 100 may instruct the adjustment reward network 130 to generate each of one or more first adjustment rewards by referring to the information on the actual actions and the actual circumstance vectors. Also, in parallel, at a step of S01-2, the learning device 100 may instruct the common reward module 170 to generate each of one or more first common rewards by referring to the information on the actual actions and the actual circumstance vectors. Further, also in parallel, at a step of S01-3, the learning device 100 may instruct the estimation network 140 to generate each of one or more actual prospective values by referring to the actual circumstance vectors.
    [0065] Thereafter, at a step of S02, the learning device 100 may instruct the first loss layer 150 to generate at least one adjustment reward loss by referring to (i) each of first personalized rewards corresponding to each of the first adjustment rewards and each of the first common rewards and (ii) the actual prospective values. Then, at a step of S03, the learning device 100 may instruct the first loss layer 150 to perform backpropagation by referring to the adjustment reward loss, to thereby learn at least part of parameters of the adjustment reward network 130.
    [0066] More specifically, each of the first adjustment rewards, generated at the step of S01-1, may be each of adjustment rewards corresponding to each of the actual actions performed at each of the timings. It was noted as the "first" adjustment rewards to distinguish those from other adjustment rewards, e.g., second adjustment rewards to be used for training the estimation network 140.
    [0067] Also, each of the first common rewards may be each of common rewards corresponding to each of the actual actions performed at each of the timings. It was also noted as the "first" common rewards to distinguish those from other common rewards, e.g., second common rewards to be used for training the estimation network 140.
    [0068] Such first adjustment rewards and such first common rewards may be summed up correspondingly to generate the first personalized rewards. By summing up those two kinds of rewards, the scheme of the present disclosure as shown below, which is adjusting the common reward function to generate the personalized reward function, can be fulfilled.
    [0069] And, the actual prospective values may be values on personalized rewards corresponding to each of the common optimal actions performed in each of the actual circumstances at each of the timings. As an example, the actual prospective values may be sums of such personalized rewards.
    [0070] Below, how the adjustment reward loss can be generated by referring to the first personalized rewards and the actual prospective values will be explained, by referring to a following formula.
    [0071] In the formula,
    [0072] More specifically, the max operation is built as shown above to (i) compare the specific actual prospective value Vcommon ( St ) which represents a sum of personalized rewards generated when the common optimal rewards are performed and its corresponding sum of personalized rewards,∑ r = t T − 1 γ r − t R commons r a r s r + 1+ R drivers r a r s r + 1 ,
    [0073] A latter part of the formula for the adjustment reward loss without the max function has been added to prevent the first adjustment rewards from becoming too large. The adjustment reward should be prevented from becoming too large, because, if it becomes too large, the personalized rewards may be overfitted to the subject driver. If the adjustment reward loss excludes the latter part, the adjustment reward network 140 may be trained to generate the adjustment rewards, and further personalized rewards, larger when its corresponding RL agent performs similarly only to the actual actions. Thus, to prevent the overfitting, the sum of absolute values of first specific adjustment rewards, among the first adjustment rewards is added to the adjustment reward loss.
    [0074] As the adjustment reward loss is generated, the first loss layer 150 may perform the backpropagation by using the adjustment reward loss, to learn the parameters thereof.
    [0075] Thereafter, the learning device 100 may perform training process of the estimation network 140. Below, it will be explained.
    [0076] That is, the learning device 100 may instruct the adjustment reward network 130 to generate one or more second adjustment rewards by referring to the actual circumstance vectors. And, in parallel, the learning device 100 may instruct the common reward module 170 to generate one or more second common rewards by referring to the actual circumstance vectors. Also, in parallel, the learning device 100 may instruct the estimation network 140 to generate one or more virtual prospective values corresponding to the virtual circumstances by referring to each of the virtual circumstance vectors. Then, the learning device 100 may instruct the second loss layer 160 to generate at least one estimation loss by referring to (i) each of second personalized rewards corresponding to each of the second adjustment rewards and each of the second common rewards, (ii) the virtual prospective values and (iii) the actual prospective values, and to perform backpropagation by referring to the estimation loss, to thereby learn at least part of parameters of the estimation network 140.
    [0077] Herein, the second adjustment rewards may represent adjustment rewards corresponding to each of the common optimal actions to be performed at each of the timings of the driving trajectories. Different from the first ones, second ones are for the common optimal actions, not for the actual actions. Also, the second common rewards may represent common rewards corresponding to each of the common optimal actions to be performed at each of the timings of the driving trajectories. Those are also for the common optimal actions, not the actual actions, different from the first ones. Therefore, the second personalized rewards generated by correspondingly summing up the second adjustment rewards and the second common rewards may correspond to personalized rewards for the common optimal actions, not the actual actions. The reason why such second personalized rewards for the "common" optimal actions are used is that the estimation network 140 is a network which estimates a sum of personalized rewards generated for the common optimal actions. Thus the second personalized rewards for the common optimal actions are used as the training data.
    [0078] By referring to the above explanation, how the estimation loss can be generated by using the second personalized rewards, the virtual prospective values and the actual prospective values will be explained, by referring to a following formula.
    [0079] Herein,
    [0080] More specifically, both R commons t a t ʹ s t + 1 ʹ + R drivers t a t ʹ s t + 1 ʹ +
    [0081] So far the training processes of the adjustment reward network 130 and the estimation network 140 have been explained. As seen in the training processes, those two networks need each other to train themselves. That is, when training the adjustment reward network 130, the actual prospective values, which are outputs of the estimation network 140, are needed, and when training the estimation network 140, the second adjustment rewards, which are outputs of the adjustment reward network 130, are needed. Thus, the two networks can be trained in turns. That is, the adjustment reward network 130 may be trained first, then the estimation network 140 may be trained, again the adjustment reward network 130 may be trained, still again the estimation network 140 may be trained, and so on. It has been illustrated that the adjustment reward network 130 is trained earlier than the estimation network 140, but such order is not an essential one, thus the estimation network 140 may be trained first.
    [0082] Herein, the training data, i.e., the driving trajectories for the subject driver, used for training the two networks, may have been provided to the learning device 100 as a mini batch by (i) transmitting a query to a database, (ii) randomly sampling the driving trajectories from a driving trajectory group corresponding to the subject driver in the database, and (iii) delivering those to the learning device 100.
    [0083] In the above explanations, it has been explained that the same mini batch is used for training both the adjustment reward network 130 and the estimation network 140, but a scope of the present disclosure may not be limited thereto. That is, each of different mini batches may be selected for each of the training processes of each of the two networks. Such tweak on the method of the present disclosure will be obvious for a person in the art, thus such embodiment is included in the scope of the present disclosure.
    [0084] So far the learning method of the present disclosure has been explained. Below, a testing method of the present disclosure will be explained.
    [0085] First, on condition that (1) the learning device 100, if one or more actual circumstance vectors for training and information on one or more actual actions for training performed at timings for training corresponding to the actual circumstance vectors for training by referring to one or more actual circumstances for training, corresponding thereto, included in each of one or more driving trajectories for training of the subject driver are acquired, has performed (i) a process of instructing the adjustment reward network 130, which is built to operate as an adjustment reward function to be used for establishing the personalized reward function from a common reward function corresponding to the common optimal policy, to generate each of one or more first adjustment rewards for training corresponding to each of the actual actions for training performed at each of the timings for training, by referring to the information on the actual actions for training and the actual circumstance vectors for training, (ii) a process of instructing the common reward module 170 corresponding to the common reward function to generate each of one or more first common rewards for training corresponding to each of the actual actions for training performed at each of the timings for training, by referring to the information on the actual actions for training and the actual circumstance vectors for training, and (iii) a process of instructing the estimation network 140, which is capable of estimating sums of personalized rewards for training generated while common optimal actions for training according to the common optimal policy for training are performed based on its corresponding actual circumstances for training, to generate each of one or more actual prospective values for training corresponding to each of the actual circumstances for training at each of the timings for training in the driving trajectories for training, by referring to the actual circumstance vectors for training; and (2) the learning device 100 has instructed the first loss layer 150 to generate at least one adjustment reward loss by referring to (i) each of first personalized rewards for training corresponding to each of the first adjustment rewards for training and each of the first common rewards for training and (ii) the actual prospective values for training, and to perform backpropagation by referring to the adjustment reward loss, to thereby learn at least part of parameters of the adjustment reward network, a testing device may instruct the adjustment reward network 130 and the common reward module 170 to generate a personalized reward for testing including an adjustment reward for testing and a common reward for testing by referring to (i) an actual circumstance vector for testing corresponding to a t-th timing and (ii) an actual action for testing generated by the personalized RL agent.
    [0086] Then, the testing device may instruct the personalized RL agent to train its own parameters by referring to the personalized reward for testing. How the RL agent uses the personalized reward function established by the common reward module 170 and the adjustment reward network 130 may be similar to that of conventional arts on the RL, thus more explanation is omitted.
    [0087] By using the personalized reward for testing to train the personalized RL agent, the subject vehicle may become capable of driving similarly to the actual actions for training, to thereby provide a driver-specific better driving experience to the subject driver autonomously.
    [0088] It is an advantage of the present disclosure to provide a learning method for supporting a Reinforcement Learning(RL) algorithm by using human driving data as training data, to thereby provide a personalized path planning, and then to thereby provide a satisfied driving experience to passengers of an autonomous vehicle. And, it is another advantage of the present disclosure to provide a personalized reward function to be used for supporting the RL algorithm, by using the human driving data as the training data, to thereby provide the personalized path planning. Further, it is still another advantage of the present disclosure to provide a method for acquiring the personalized reward function by adjusting a common reward function to reduce usage of computing resources.
    [0089] The embodiments of the present disclosure as explained above can be implemented in a form of executable program command through a variety of computer means recordable to computer readable media. The computer readable media may include solely or in combination, program commands, data files, and data structures. The program commands recorded to the media may be components specially designed for the present disclosure or may be usable to a skilled human in a field of computer software. Computer readable media include magnetic media such as hard disk, floppy disk, and magnetic tape, optical media such as CD-ROM and DVD, magneto-optical media such as floptical disk and hardware devices such as ROM, RAM, and flash memory specially designed to store and carry out program commands. Program commands include not only a machine language code made by a complier but also a high level code that can be used by an interpreter etc., which is executed by a computer. The aforementioned hardware device can work as more than a software module to perform the action of the present disclosure and they can do the same in the opposite case.
    [0090] As seen above, the present disclosure has been explained by specific matters such as detailed components, limited embodiments, and drawings. They have been provided only to help more general understanding of the present disclosure. It, however, will be understood by those skilled in the art that various changes and modification may be made from the description without departing from the spirit and scope of the disclosure as defined in the following claims.
    [0091] Accordingly, the thought of the present disclosure must not be confined to the explained embodiments, and the following patent claims as well as everything including variations equal or equivalent to the patent claims pertain to the category of the thought of the present disclosure.
    权利要求:
    Claims (15)
    [0001] A learning method for acquiring at least one personalized reward function, used for performing a Reinforcement Learning(RL) algorithm, corresponding to a personalized optimal policy for a subject driver which is acquired by adjusting a common optimal policy established according to a common criterion for an autonomous driving, to thereby support the autonomous driving of a subject vehicle, comprising steps of:
    (a) a learning device, if one or more actual circumstance vectors and information on one or more actual actions performed at timings corresponding to the actual circumstance vectors by referring to one or more actual circumstances, corresponding thereto, included in each of one or more driving trajectories of the subject driver are acquired, performing (i) a process of instructing an adjustment reward network, which is built to operate as an adjustment reward function to be used for establishing the personalized reward function from a common reward function corresponding to the common optimal policy, to generate each of one or more first adjustment rewards corresponding to each of the actual actions performed at each of the timings, by referring to the information on the actual actions and the actual circumstance vectors, (ii) a process of instructing a common reward module corresponding to the common reward function to generate each of one or more first common rewards corresponding to each of the actual actions performed at each of the timings, by referring to the information on the actual actions and the actual circumstance vectors, and (iii) a process of instructing an estimation network, which is capable of estimating sums of personalized rewards generated while common optimal actions according to the common optimal policy are performed based on its corresponding actual circumstances, to generate each of one or more actual prospective values corresponding to each of the actual circumstances at each of the timings in the driving trajectories, by referring to the actual circumstance vectors; and
    (b) the learning device instructing a first loss layer to generate at least one adjustment reward loss by referring to (i) each of first personalized rewards corresponding to each of the first adjustment rewards and each of the first common rewards and (ii) the actual prospective values, and to perform backpropagation by referring to the adjustment reward loss, to thereby learn at least part of parameters of the adjustment reward network.
    [0002] The method of Claim 1, wherein, at the step of (b), the learning device instructs the first loss layer to generate the adjustment reward loss by referring to a following formula:
    [0003] The method of Claim 1, further comprising steps of:
    (c) the learning device performing (i) a process of instructing the adjustment reward network to generate one or more second adjustment rewards corresponding to each of the common optimal actions to be performed at each of the timings of the driving trajectories by referring to the actual circumstance vectors, (ii) a process of instructing the common reward module to generate one or more second common rewards corresponding to each of the common optimal actions to be performed at each of the timings of the driving trajectories by referring to the actual circumstance vectors, and (iii) a process of instructing the estimation network, by referring to each of one or more virtual circumstance vectors corresponding to each of virtual circumstances caused by performing the common optimal actions at each of the timings of the driving trajectories, to generate one or more virtual prospective values corresponding to the virtual circumstances; and
    (d) the learning device instructing a second loss layer to generate at least one estimation loss by referring to (i) each of second personalized rewards corresponding to each of the second adjustment rewards and each of the second common rewards, (ii) the virtual prospective values and (iii) the actual prospective values, and to perform backpropagation by referring to the estimation loss, to thereby learn at least part of parameters of the estimation network.
    [0004] The method of Claim 3, wherein, at the step of (d), the learning device instructs the second loss layer to generate the estimation loss by referring to a following formula:
    [0005] The method of Claim 3, wherein the virtual circumstance vectors are acquired by applying a circumstance estimation operation to each of at least part of the actual circumstance vectors and its corresponding common optimal action corresponding to the common optimal policy,wherein the circumstance estimation operation is performed by a pre-trained circumstance expectation network or performed by (i) instructing a virtual world simulator to simulate a specific actual circumstance corresponding to a specific actual circumstance vector in a virtual world, (ii) instructing a virtual vehicle in the specific actual circumstance to perform one of the common optimal actions according to the common optimal policy, and (iii) detecting changes of the virtual world caused by said one of the common optimal actions.
    [0006] The method of Claim 3, wherein the learning device repeats a process of training the adjustment reward network corresponding to the step of (a) and the step of (b) and a process of training the estimation network corresponding to the step of (c) and the step of (d), to thereby fully train the adjustment reward network and the estimation network.
    [0007] The method of Claim 1, wherein the driving trajectories are provided to the learning device as a mini batch generated by randomly sampling the driving trajectories from a driving trajectory group corresponding to the subject driver.
    [0008] The method of Claim 1, wherein the common optimal actions according to the common optimal policy are determined by a general RL agent optimized by performing the RL algorithm using the common reward module corresponding to the common optimal policy.
    [0009] A testing method for using at least one personalized reward function to train a personalized Reinforcement Learning(RL) agent, corresponding to a personalized optimal policy for a subject driver which is acquired by adjusting a common optimal policy established according to a common criterion for an autonomous driving, to thereby support the autonomous driving of a subject vehicle, comprising steps of:
    (a) on condition that (1) a learning device, if one or more actual circumstance vectors for training and information on one or more actual actions for training performed at timings for training corresponding to the actual circumstance vectors for training by referring to one or more actual circumstances for training, corresponding thereto, included in each of one or more driving trajectories for training of the subject driver are acquired, has performed (i) a process of instructing an adjustment reward network, which is built to operate as an adjustment reward function to be used for establishing the personalized reward function from a common reward function corresponding to the common optimal policy, to generate each of one or more first adjustment rewards for training corresponding to each of the actual actions for training performed at each of the timings for training, by referring to the information on the actual actions for training and the actual circumstance vectors for training, (ii) a process of instructing a common reward module corresponding to the common reward function to generate each of one or more first common rewards for training corresponding to each of the actual actions for training performed at each of the timings for training, by referring to the information on the actual actions for training and the actual circumstance vectors for training, and (iii) a process of instructing an estimation network, which is capable of estimating sums of personalized rewards for training generated while common optimal actions for training according to the common optimal policy for training are performed based on its corresponding actual circumstances for training, to generate each of one or more actual prospective values for training corresponding to each of the actual circumstances for training at each of the timings for training in the driving trajectories for training, by referring to the actual circumstance vectors for training; and (2) the learning device has instructed a first loss layer to generate at least one adjustment reward loss by referring to (i) each of first personalized rewards for training corresponding to each of the first adjustment rewards for training and each of the first common rewards for training and (ii) the actual prospective values for training, and to perform backpropagation by referring to the adjustment reward loss, to thereby learn at least part of parameters of the adjustment reward network, a testing device instructing the adjustment reward network and the common reward module to generate a personalized reward for testing including an adjustment reward for testing and a common reward for testing by referring to (i) an actual circumstance vector for testing corresponding to a t-th timing and (ii) an actual action for testing generated by the personalized RL agent;
    (b) the testing device instructing the personalized RL agent to train its own parameters by referring to the personalized reward for testing.
    [0010] The method of Claim 9, wherein, at the step of (b), the personalized RL agent trains said its own parameters by referring to the personalized reward for testing, to thereby support the subject vehicle to drive similarly to the actual actions for training.
    [0011] A learning device for acquiring at least one personalized reward function, used for performing a Reinforcement Learning(RL) algorithm, corresponding to a personalized optimal policy for a subject driver which is acquired by adjusting a common optimal policy established according to a common criterion for an autonomous driving, to thereby support the autonomous driving of a subject vehicle, comprising:
    at least one memory that stores instructions; and
    at least one processor configured to execute the instructions to perform processes of: (I) if one or more actual circumstance vectors and information on one or more actual actions performed at timings corresponding to the actual circumstance vectors by referring to one or more actual circumstances, corresponding thereto, included in each of one or more driving trajectories of the subject driver are acquired, performing (i) a process of instructing an adjustment reward network, which is built to operate as an adjustment reward function to be used for establishing the personalized reward function from a common reward function corresponding to the common optimal policy, to generate each of one or more first adjustment rewards corresponding to each of the actual actions performed at each of the timings, by referring to the information on the actual actions and the actual circumstance vectors, (ii) a process of instructing a common reward module corresponding to the common reward function to generate each of one or more first common rewards corresponding to each of the actual actions performed at each of the timings, by referring to the information on the actual actions and the actual circumstance vectors, and (iii) a process of instructing an estimation network, which is capable of estimating sums of personalized rewards generated while common optimal actions according to the common optimal policy are performed based on its corresponding actual circumstances, to generate each of one or more actual prospective values corresponding to each of the actual circumstances at each of the timings in the driving trajectories, by referring to the actual circumstance vectors; and (II) instructing a first loss layer to generate at least one adjustment reward loss by referring to (i) each of first personalized rewards corresponding to each of the first adjustment rewards and each of the first common rewards and (ii) the actual prospective values, and to perform backpropagation by referring to the adjustment reward loss, to thereby learn at least part of parameters of the adjustment reward network.
    [0012] The device of Claim 11, wherein, at the process of (II), the processor instructs the first loss layer to generate the adjustment reward loss by referring to a following formula:
    [0013] The device of Claim 11, wherein the processor further performs a process of:(III) performing (i) a process of instructing the adjustment reward network to generate one or more second adjustment rewards corresponding to each of the common optimal actions to be performed at each of the timings of the driving trajectories by referring to the actual circumstance vectors, (ii) a process of instructing the common reward module to generate one or more second common rewards corresponding to each of the common optimal actions to be performed at each of the timings of the driving trajectories by referring to the actual circumstance vectors, and (iii) a process of instructing the estimation network, by referring to each of one or more virtual circumstance vectors corresponding to each of virtual circumstances caused by performing the common optimal actions at each of the timings of the driving trajectories, to generate one or more virtual prospective values corresponding to the virtual circumstances; and (IV) instructing a second loss layer to generate at least one estimation loss by referring to (i) each of second personalized rewards corresponding to each of the second adjustment rewards and each of the second common rewards, (ii) the virtual prospective values and (iii) the actual prospective values, and to perform backpropagation by referring to the estimation loss, to thereby learn at least part of parameters of the estimation network.
    [0014] The device of Claim 13, wherein, at the process of (IV), the processor instructs the second loss layer to generate the estimation loss by referring to a following formula:
    [0015] The device of Claim 13, wherein the virtual circumstance vectors are acquired by applying a circumstance estimation operation to each of at least part of the actual circumstance vectors and its corresponding common optimal action corresponding to the common optimal policy,wherein the circumstance estimation operation is performed by a pre-trained circumstance expectation network or performed by (i) instructing a virtual world simulator to simulate a specific actual circumstance corresponding to a specific actual circumstance vector in a virtual world, (ii) instructing a virtual vehicle in the specific actual circumstance to perform one of the common optimal actions according to the common optimal policy, and (iii) detecting changes of the virtual world caused by said one of the common optimal actions.
    类似技术:
    公开号 | 公开日 | 专利标题
    Sallab et al.2016|End-to-end deep reinforcement learning for lane keeping assist
    Kuefler et al.2017|Imitating driver behavior with generative adversarial networks
    Codevilla et al.2018|End-to-end driving via conditional imitation learning
    Williams et al.2017|Information theoretic MPC for model-based reinforcement learning
    Sadigh et al.2016|Planning for autonomous cars that leverage effects on human actions.
    Chen et al.2020|Learning by cheating
    Kahn et al.2018|Self-supervised deep reinforcement learning with generalized computation graphs for robot navigation
    Bahram et al.2016|A combined model-and learning-based framework for interaction-aware maneuver prediction
    US9104186B2|2015-08-11|Stochastic apparatus and methods for implementing generalized learning rules
    Galceran et al.2017|Multipolicy decision-making for autonomous driving via changepoint-based behavior prediction: Theory and experiment
    Dong et al.2017|Intention estimation for ramp merging control in autonomous driving
    Schmerling et al.2018|Multimodal probabilistic model-based planning for human-robot interaction
    Rhinehart et al.2019|Precog: Prediction conditioned on goals in visual multi-agent settings
    Sharifzadeh et al.2016|Learning to drive using inverse reinforcement learning and deep q-networks
    Ono et al.2015|Risk-aware planetary rover operation: Autonomous terrain classification and path planning
    Pfeiffer et al.2018|Reinforced imitation: Sample efficient deep reinforcement learning for mapless navigation by leveraging prior demonstrations
    Bandyopadhyay et al.2013|Intention-aware pedestrian avoidance
    Agamennoni et al.2012|Estimation of multivehicle dynamics by considering contextual information
    DE102016113903A1|2017-03-02|Vehicle route determination
    CN107697070B|2020-04-07|Driving behavior prediction method and device and unmanned vehicle
    Rhinehart et al.2018|Deep imitative models for flexible inference, planning, and control
    US20180253647A1|2018-09-06|Offline combination of convolutional/deconvolutional and batch-norm layers of convolutional neural network models for autonomous driving vehicles
    US10324469B2|2019-06-18|System and method for controlling motion of vehicle in shared environment
    US10678253B2|2020-06-09|Control systems, control methods and controllers for an autonomous vehicle
    Ratliff et al.2007|Boosting structured prediction for imitation learning
    同族专利:
    公开号 | 公开日
    CN111507501A|2020-08-07|
    US20200250486A1|2020-08-06|
    JP2020126646A|2020-08-20|
    KR20200095378A|2020-08-10|
    引用文献:
    公开号 | 申请日 | 公开日 | 申请人 | 专利标题
    法律状态:
    2020-07-03| PUAI| Public reference made under article 153(3) epc to a published international application that has entered the european phase|Free format text: ORIGINAL CODE: 0009012 |
    2020-07-03| STAA| Information on the status of an ep patent application or granted ep patent|Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
    2020-08-05| 17P| Request for examination filed|Effective date: 20200121 |
    2020-08-05| AK| Designated contracting states|Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
    2020-08-05| AX| Request for extension of the european patent|Extension state: BA ME |
    2021-03-10| RBV| Designated contracting states (corrected)|Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
    2021-07-30| STAA| Information on the status of an ep patent application or granted ep patent|Free format text: STATUS: EXAMINATION IS IN PROGRESS |
    2021-09-01| 17Q| First examination report despatched|Effective date: 20210729 |
    优先权:
    申请号 | 申请日 | 专利标题
    [返回顶部]